Bayesian networks with JAGS

Network structures and models

Structure

Models

\[ \text{T1D | Metabolite} \sim \text{Bern}(\text{logit}(\alpha_0 + \alpha_1*\text{Metabolite}))\\ \text{Methylation | Metabolite, T1D} \sim \text{N}(\delta+\gamma*\text{Metabolite}+\beta*\text{T1D},\sigma^2)\\ \]

Priors

\[ \alpha_0 \sim \text{N}(0,100)\\ \alpha_1 \sim \text{N}(0,100)\\ \delta \sim \text{N}(0,100)\\ \gamma \sim \text{N}(0,100)\\ \beta \sim \text{N}(0,100)\\ \sigma^2 = \frac{1}{\tau}\\ \tau \sim \text{Gamma}(0.0001,0.0001) \]

Model comparisons

DIC distribution by network structure

Cit-selected pairs

Number of Pairs Where Structure Is Best
Var1 Freq
10 8
14 94
21 1
23 2
24 34
Number of Pairs Where Structure Is Best or Within 2 DIC
Var1 Freq
10 11
14 128
21 6
23 5
24 64

Non-cit pairs

Number of Pairs Where Structure Is Best
Var1 Freq
14 4
19 43
21 8
23 2
24 82
Number of Pairs Where Structure Is Best or Within 2 DIC
Var1 Freq
10 8
14 129
19 105
21 10
23 10
24 120

Permutation tests

This feels like more of a sanity check that the permutations tests are working, but I’m not sure if it answers our questions about picking the best model.

Potential problems

DIC

  • “From Bayesian perspective, DIC is not theoretically justified since it measures the fit of the model when the parameters are fixed to the posterior expectation and is not therefore an unbiased estimate of the true generalization utility.” (Piironen & Vehtari, 2017)

  • “The numerical experiments show that the over-fitting in the selection may be a potential problem and hinder the model selection considerably. This is the case especially when the dataset is small (high variance in the utility estimates) and the number of models under comparison large (large number of variables). Especially vulnerable methods for this type of overfitting are CV, WAIC, DIC and other methods that rely on data reuse and have therefore relatively high variance in the utility estimates.” (Piironen & Vehtari, 2017)

Priors

Tim was wrong about gamma priors causing problems. This issue is specific to logistic models with random effects, where the prior is \(b_i \sim N(0,\frac{1}{\lambda})\). The Piironen paper can be a little unclear, so maybe we need a better resource.

References